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ABSTRACT 

The procedures used in the initial development of a 
Mathematics Laboratory within the Individually Prescribed Instruction 
(IPI) Mathematics Program are reported. The thesis of the report is 
that in the development of educational programs, especially those 
which are complex, attention should be directed both to the design of 
the program and the design of its evaluation, both formative and 
s umma tive. Using the Lindva 11-Cox model framework,, dt is suggested 
that one product of each phase of the development effort should be a 
revised “Goals- Plan-Operation-Assessment" outline, with such an 
outline then being used in the subsequent phase to guide its 
installation and evaluation. The Evaluation Outline presented 
contains in a concise form a description of the program at this state 
of its development. (Author/DB) 
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The Use of Formative Evaluation Procedures in the 
Development of a Mathematics Laboratory*^* 



George R. Miller 
University of Pittsburgh 



In his frequently quoted 1963 article, "Course Improvement Through 
Evaluation," Cronbach (1963) developed the thesis that evaluation can make 
its most important* contribution to the improvement of education when it 
is used as a basis for program revision rather than for any type of final 
assessment. Shortly thereafter, Scriven (1967) used the term "formative 
evaluation" to differentiate the role of evaluation in program development 
from its "summative" role in the final assessment of a program. Stake (1969) 
distinguished between these two roles - one providing the information the 
"program people" want to know and the other what "outsiders'* want to know. 

He states : 

"We can make a non-trivial distinction between 
formative evaluation for the program developer 
who is planning ahead and trying to choose the 
best ingredients, and summative evaluation for 
anyone who is looking at the program, past or 
present, and who is trying to find out what it 
is and what it does." (1969, p. 40) 

In attempting to more fully describe these roles of evaluation, 
a number of persons have proposed models and procedures for the systematic 
planning and carrying out of a total evaluation program. These models 
place major emphasis on the integration of evaluation and program develop- 
ment activities* Stuff lebeam (1968) describes four cyclic evaluation 
stages through which a developer could proceed to make necessary 

^Appreciation is expressed to Millie K. Sass, who assisted in the 
implementation of this laboratory. 
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decisions concerning his program. These evaluation stages are: (1) Context - 

determine objectives;. (2) Input - design procedures; (3) Process - 
utilize control and refine procedures; (4) Product - judge and react 
to attainments. Stake (1967) identifies six stages and uses the term 
"congruence" to relate what is intended to what actually is in operation. 

In a somewhat comparable model, but one which is directed more 
to the process of developing a new educational program, Lindvall and 
Cox (1970) outline formative evaluation activities that permit the developer 
to examine each element in the design and operation of the program. They 
suggest that such an examination will be enhanced if the program is out- 
lined using four categories which: (1) specify the program’s Goals ; 

(2) outline a Plan for the achievement of these goals; (3) describe 
the plan as it should appear in Operation ; and (4) specify the Assessment 
procedures that will be used to measure the achievement of the established 
goals. 

Figure 1 specifies the formative evaluation activities that 
should be performed in utilizing such an outline in the development of a 
program. This evaluation focuses on four basic questions: 

(1) What goals should the program achieve? 

(2) What is the plan for achieving these goals? 

(3) Does the operating plan represent a true implementation 
of the plan? 

(4) Does the operating plan achieve the desired goals? 

The two-way loops below each program category describe the necessary 
procedures that should be performed in the answering of these questions. 



2 3 

O 

ERJC 



* 



As these loops Illustrate, this formative evaluation process must examine 
how each step Is dependent upon the step or steps that precede It and how 
each Influences those which follow. 

Purpose of Study 

This study, performed within the same context In which the Lindvall 
Cox model was developed, the Individually Prescribed Instruction (IPI) 
Project of the Learning Research and Development Center at the University 
of Pittsburgh, is a description of the development process, when formative 
evaluation is taken quite seriously, in the installation of a new program 
component. Specifically, this study is a report of the procedures used in 
the initial development of a Mathematics Laboratory within the IPI 
Mathematics Program. 

The thesis of this paper is that in the development of educational 
programs, especially those which are complex, attention should be directed 
both to the design of the program and the design of its evaluation - both 
formative and summative. That is, in addition to testing and refining the 
operating plan, the developer should, at the same time, be defining 
procedures and strategies which will provide him in the future with valid 
information as to when the operation reflects the plan and the degree of 
attainment of the program's goals. 

Using the Lindvall-Cox framework, this suggests that one product 
of each phase of the development effort should be a revised "Goals-Plan- 
Operation-Assessment" outline. Such an outline would then be used in the 
subsequent phase to guide its installation and evaluation. 
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The characteristics of the laboratory, as tested in this study, 
will be presented in the following description of the steps followed in 
its initial tryout in a classroom. This paper will primarily focus on: 
(1) the methodology used to formulate the evaluation outline prior to 
the tryout; (2) a description and evaluation of the tryout; and (3) the 
formulation of the "Goals-Plan-Operation-Assessment" outline for the 
next development phase. 



Development - Phase I 

Specification of Goals and Development of Plan 

The specific goals of the lab were derived from an assessment of 
the existing IPI Math Program and a consideration of the desired additional 
learning experiences that might be provided by such a learning setting. 
These goals (see Figure 2), stated in terms of observable pupil behavior, 
describe the kinds of learning experiences which were viewed at this time 
to be the ones a pupil should have when the lab is functioning in final 
form. A plan to accomplish these goals was then defined. 

This initial plan is based on the best judgement of members of 
the development staff, who make use of all available sources of knowledge 
concerning what can be expected to be effective in such a program. The 
above involves such things as searching the relevant literature, inter- 
viewing teachers and other practitioners, and drawing on the past 
experiences of the staff itself. 

The elements of the Phase I Plan are listed in the right hand 
columns of Figure 2, under the headings: Instructional Objectives, 



Pupil Evaluation Procedures, Instructional Activities, Classroom Management 
Procedures, Teacher Activities, and Pupil Activities. The initial evalu- 
ation of this plan was carried out through logical analysis of the 
relationship of each element in the plan to one of the goals of the lab. 

That is, concerning each element, the question was raised, "Does this 
have promise for helping to achieve one or more of the program's goals?" 

In the model employed in the present study, this planning is taken 
through several phases of a "plan-operation-assessment" cycle. Phase I 
involves a limited tryout and assessment of the initial plan. This phase 
may be compared with the task of the person who is developing a unit in a 
programmed textbook when he tries out his initial program with only one or 
two students. The purpose is to get ideas that will be useful in the further 
development of his first draft . The information used by the programmer is 
obtained by watching the pupil as he studies the program, by studying his 
responses, and by interviewing the pupil. In applying a comparable 
procedure in the development of this lab, the initial plan is tried out 
with one or two groups of students, the operation is closely observed, 
pupil activities and verbal responses are noted, and the teacher is inter- 
viewed. This type of informal, but highly informative, assessment informa- 
tion is then used in revising the plan for Phase II. This initial procedure # 
was described earlier as an extension of the Lindvall-Cox Model, in that it 
calls for carrying out the "goals-plan-operation-assessment" cycle at 
several successive stages of program development. How this close observation 
of the first tryout of this initial plan was actually carried out is 
described in the following section. 
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Operation - Phase I 



The setting for the Phase I implementation was in one first grade 
class in an IPI experimental school during the Spring of 1971. The labora- 
tory was contained in the classroom and pupils were assigned to it during 
their regular mathematics period. In order to gain maximum information 
from this tryout, two project personnel managed the laboratory in place of 
the regular teacher - one acting as the "teacher," the other as an "observer 
The math content taught in this tryout was limited to activities in 
Beginning Linear Measurement, a unit in which the pupils had not had pre- 
vious formal instruction. (See Figure 3 for objectives of this unit). 

The tryout setting permitted tiue assignment of pupils to the labora- 
tory in groups of three pupils each day. The first session for each group 
. was conducted by a Project Staff member. In this session, the pupils were 
told how to use the lessons, where to get materials, and how to record 
results in their notebooks. They then worked through one’ of the lessons 
under the supervision of this person. They were directed to estimate 
before they measured and to check the accuracy of each measurement by 
replication or by requesting another pupil to measure the same object and 
to compare their results. In addition, the pupils were directed not to ask 
the teacher for help until they had requested assistance from another 
member of the lab. 

The lab activities were presented on laminated cards, using pictures 
and written instructions involving a vocabulary thought to be appropriate 
for this age pupil. These activities placed the pupil in situations in 
which he could learn the measurement outcomes of the unit, as well as 
offer him a degree of choice in what he was to measure. 



Figure 4 is an example of these cards. This is one of several 
activities in which the pupil uses individual "inch" and "foot" rods 
to measure in preparation for the use of a standard ruler. The purpose of 
this lesson is to show that the length of an object can be stated in 
more than one way, as well as affording the opportunity to "discover" that 
12 inches is equivalent to one foot. 

The same activity was given to each member of the three-pupil group. 
Each was reminded to read the card and to ask help of his "neighbor" if he 
needed assistance and to ask the teacher only as a last resort. As a 
"rule-of-thumb," a pupil was refused help two times on a given problem 
before the teacher intervened. The three-pupil-per-day assignment made 
it possible to revise a lesson on the basis of observed problems and then 
use the revised lesson with the next day's group. When all pupils had 
completed one "cycle," the next appropriate activity in the sequence was 
studied. Time permitted only five of such "cycles" in the ten weeks. In 
the third through fifth cycles, members of the groups received different 
cards in order to test more lessons. 

This tryout of the effectiveness of lesson materials depended upon 
the pupils acquiring and maintaining a degree of independence from the 
teacher. In the initial cycles of this tryout, teacher influence was 
directed to shaping this behavior through, (1) refusal to answer questions 
which the teacher felt could be answered by the pupil or by others in the 
group, and (2) by verbal rewards for self-directed activity. Also, 
attention was given to the problem of how the teacher could best interact 
with pupils to promote discovery and exploration, in addition to that 
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provided by the lessons. The effectiveness of guiding questions and 
brief demonstrations was explored to develop this behavior. 

In the fourth and fifth cycles the regular classroom teacher 
managed and controlled some lab sessions in addition to her regular class 
individualized instruction. Prior to each class period, she was directed 
to perform certain interactions with the lab pupils. The observer noted 
these interactions and their effect on pupils after she had left. The 
teacher also provided important information missed by the observers, due 
no doubt to her familiarity with the pupils. 

Evaluation of Operation - Phase I 

A key focus of formative evaluation is in examining a program 
when it is first placed in operation. Here the intent is both to deter- 
mine the extent to which the planned program is actually implemented and 
to assess the extent to which program goals are being approximated. In 
evaluating the operation of the laboratory, an observer was stationed in 
the room throughout the tryout. He recorded in diary form the activities 
of each day. After each session, the project personnel met and discussed 
the days experiences. Those events which were thought to be pertinent to 
the program's development and evaluation were noted. No formal observa- 
tion instruments were used. 

The results of this evaluation of the operation in the Phase I 
Cycle can best be discussed in relation to each category of the plan. 

Instructional Objectives . The performance objectives, stated in 
terms of the observable pupil behavior were those included in the beginning 
linear measurement units of the IPI Continuum. They were found to be 
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necessary in the design of the lesson activities so that the measuring 
experiences would focus on the desired pupil outcomes. 

Pupil Evaluation Procedures . The plan elements under this category 
reflect "self-evaluation" components which are thought to be required when 
the pupil was "self-managing" his work in the lab. Since these elements 
were not implemented in this tryout, their evaluation cannot be made at 
this time. 

Instructional Activities . The type of changes made in the activity 
cards were too many to enumerate in this paper. The modifications were 
mainly in format and appearance of the card, type and wording of questions, 
and general organization. It was found that undue novelty and complexity 
of operations should be avoided, at least for this age pupil (6-7 years). 
Examples of a process or a diagram of recording procedures were useful to 
some pupils. Questions should be simple and prominently arranged on the 
card. Numbering of steps or questions should be avoided, unless they are 
useful in the demonstration of a multiple-step process. Recording of re- 
sults was facilitated when an established format was used in the notebooks 
rather than blank paper. 

Classroom Management Procedures . The lab model was designed for 
sn in— classroom operation. One concern in this tryout was whether the 
teacher, with her ether classroom duties, would have the time to make her 
necessary interactions with pupils. An additional one was the possibility 
that the pupil activity generated by this setting would be distracting to 
other pupils. It appears from this limited tryout that the teacher could 
manage these pupils, as well as her "regular" instructional duties. The 
activity of the pupils was not as distracting to others and in some cases 



proved to be a "non-planned" productive experience for lab-pupils to 
interact with other students on measurement tasks* 

Teacher Classroom Activities * Once the pupils were oriented to the 
laboratory procedures and the "rule” ("Ask someone in the lab if you need 
help"), verbal reinforcement by the teacher appeared to be sufficient to 
sustain these behaviors in most of the pupils* The teacher "traveling 
model" which influenced the most productive pupil behavior was: (1) obser- 

vation at the beginning of the period} (2) a ten to fifteen minute wait} 
then (3) short (1 minute or less) interactions through the remainder of 
the period. These interactions were generally teacher-initiated questions. 

It was noted that the teacher-initiated interactions, at times, interrupted 
productive pupil activity (as assumed by the observer). This suggests 
that the teacher should be prudent in interrupting pupils for these 
interactions . 

Pupil Classroom Activities . By the fourth and fifth cycles (the 
fourth and fifth time in the lab), a majority of the pupils had achieved a 
measure of self -directed behavior. This was evidenced by sustained attempts 
to read the activity cards, helping behavior, and proper use of the materials 
as suggested by the lessons. Checking behavior, however, was not performed 
by a majority of the pupils without constant reminders. Pupils did enjoy 
selecting their own objects to measure. 

Non-Planned Aspects of the Operation . Once a program is installed 
in such an in-context setting , many non-planned behaviors do occur . Some 
appear to be significant enough to be considered for incorporation to the 
evolving plan of this lab. One significant contribution was the ease to 
which the teacher, in her interactions with pupils, could relate math skills 




to a measurement task. Some of these first grade pupils were quite amazed 
that their knowledge of beginning arithmetic operations could be applied 
to their measuring problems. 

Assessment 

The ultimate achievement of a program's goals will be determined 

by some final assessment. However, during development, the operating plan 

must be continually assessed as to the extent to which its elements are 

contributing to this achievement. As with the evaluation in the operation 

stage described above, the assessment stage in Phase I is carried out so 

as to provide information on the achievement of program goals, as well as 

contribute to the further refinement of the plan and operation. 

Goal 1 - The pupil learns through active involvement, both independently 
and with other pupils. 

All pupils were quite active in the performance of their assigned 
measuring tasks. They recorded their estimates and measurements in their 
notebooks and some pupils, at times, stated the number of units they were 
"off" in their Estimates. The "free" environment designed in this tryout 
resulted in many instances of "pupil-arranged" cooperative behavior. 

This included assisting each other in measuring, reading lessons and 
recording in notebooks. At times, a pupil could be observed just watching 
another pupil performing a task - these observations interspersed with a 
comment or question. 

Goal 2 - The pupil selects activities for which he determines that he has 
the necessary prerequisites. 

Plan elements were not implemented to achieve this goal, in that 
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a ll activities were assigned to pupils to test lesson design procedures. 
However, the experience of this tryout suggests that for pupils to self-select 
activities for which they will have a meaningful learning experience, some 
provision must be made for them to evaluate their own competencies. 

Goal 3 - The pupil formulates new examplars of the concept. 

This objective encompasses behaviors that involve the application 
or extension of an activity to new instances defined by the pupil (and not 
specified by the lesson itself). Each activity card provided the oppor- 
tunity for the performance of this behavior, by suggesting to the pupil 
that he could measure objects of his choice in the room. The pupils 
appeared to enjoy this time and some would spend the remaining time in the 
period measuring, many times striving to locate something unique to measure. ■ 
However, the teacher's influence is needed to guide some pupils in pro- 
ductively using this opportunity to apply or extend their learning. 

Goal 4 - The pupil formulates and tests hypotheses. 

A defining characteristic of laboratory learning is that the pupil 
learns through the process of formulating a hypothesis and subsequently 
testing this conjecture using concrete materials or math skills. It is 
thought that the success of this method will vary as to abilities of the 
pupil, as well as to the topic studied. Therefore, the degree of sophis- 
tication in hypothesis formation and testing expected for first— graders 
would, no doubt, be different from that desired for sixth-grade pupils. 

In this tryout, pupils did exhibit some low level hypothesis 
formulation and testing behaviors. Practically, all pupils by the end 
of the ten weeks would estimate the measurement of an object before they 
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measured. In addition, many of the activities presented measurement 
situations which required the pupil to formulate a "method of attack" 
to arrive at the solution. "Without moving the desk, can you tell if 
it can be moved through the doorway?" This could be considered a hypo- 
thesis in the form, "I can find put if the desk fits through the door 
if I cut a piece of string the same size as the desk and compare its 
length to the width of the door." 

Other than the evidence of estimation, no other data was collected 
on the degree of attainment of this goal. It can only be assumed that in 
the act of solving problems similar to the one described above, the. child 
may have proceeded in this fashion. It is suspected that the consistent 
use of this process by pupils can best be developed through the experiences 
included in the activity cards, as well as teacher-pupil interactions 
directed to shaping this type of behavior. 

Goal 5 - The pupil monitors and evaluates his own progress. 

Plan elements were not implemented to achieve this goal. How- 
ever, in consultation with the teachers, they were of the opinion that 
the development of this behavior might be quite difficult for primary age 
children. This goal, then, should be modified to include only the recording 
of progress by pupils; the monitoring to be a task of the teacher. 

Goal 6 - The pupil selects activities of interest to him. 

The plan elements providing for pupil self-selection of activity 
cards were not implemented to achieve this goal. Pupils did have the 
opportunity to select their own objects to measure, ac described pre- 
viously. Both this Goal and Goal 2 reflect selection behavior, so they 
might well be combined. • ... 
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Goal 7 - The pupil acquires skills and comprehension by carrying out 
meaningful activities and experiments. 

The instructional effectiveness of the laboratory in this ten- 
week tryout was measured by the administration of a criterion-referenced 
test on the objectives of the unit. Using a posttest-only design, this 
test was administered to the lab-class and a companion first-grade which 
had not received any formal measurement instruction. 

The results of this test are summarized in Figure 5. This infor- 
mation showed that the lab activities and procedures did have an effect on 
some pupils in at least three of the six pupil outcomes. It also suggests 
that Objective 1 (makes gross comparisons on the basis of length, i.e., 
longest, shortest) is probably taught elsewhere in the school program and 
could possibly be eliminated from the measurement curriculum. The 
similar percentages of pupils mastering objectives 4 and 5 in the two 
groups appears to show that the materials and procedures were not effective 
in increasing the competencies of the lab group. The low percentage 
scores on the other objectives, which did show a differential effect of 
lab activities, were somewhat expected since time did not permit pupils 
to use all of the activities designed to teach these objectives. 

It must be stressed that the administration of this type of test, 
at this point in development, has essentially no implications for the 
eventual worth of the program. This information, as well as the previous 
discussion of the effect of this tryout on the other goals* has value 
only in improving this development effort. 
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This study described the development and application of a forma- 
tive evaluation procedure in the initial design of a mathematics laboratory 
for young children. Since the refinement of this design involves a re- 
cycling of steps in successive "phases of revision," the results of the 
Phase I activities can best be represented by the Goals and Plan for 
Phase II. These are specified in the outline for Phase II, as formulated 
at this point in the development and evaluation process (see Figure 6) . 

The first column lists the "Goals" for the program in Phase II. 

It will be noted that these goals do differ slightly from those proposed 
at the beginning of the project. Some have been combined and others 
reworded for clarity. Several additional goals have been proposed which 
define outcomes which are viewed at this time as being relevant to the 
rationale and management of a laboratory in relation to the IPI Math 
Program. 

The "Plan" elements define the lab at this stage of development 
In comparison with the Phase I plan, some elements have remained the same 
and others rephrased. In addition, new elements have been added as a 
result of the experience in this tryout. What is not reflected here is 
the relative importance of some of these to the achievement of their 
associated goals. 

The "Operations" column defines the observable characteristics 
of each element in the Plan. From this listing, the observation schedules 
will be prepared to monitor the Phase II implementation; this monitoring, 
of course, is fundamental in the formative evaluation of a development 
effort. 
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The "Assessment" column lists the general methods to be used to 
measure the degree to which the operating plan results in the achievement 

t 

of the stated goals. These methods are only descriptive of what will be 
done or used in the on-going assessment. 

This Evaluation Outline presents in concise form a description of 
the program at this stage of its development. The operating plan will be 
more adequately described in teacher manuals and observation schedules. 
Some of the assessment instruments are presently defined, while others 
will be devised during the next development phase when characteristics 
of goal achievement can be more adequately defined. 

This paper proposes a model of development which places emphasis 
on the design of a formative evaluation, as well as use of this information 
for program improvement. In this description of the Phase I tryout and 
the design of the laboratory for the next development phase, much of the 
information gleaned was quite subjective in nature. But, as development 
proceeds, more empirical data will be collected which can be used to refine 
the program so a reliable operational plan results in the achievement of 
the program's goals. 














References 



Cronbach, L. J. Course improvement through evaluation. Teachers 
College Record (1963), May. 

Lindvall, C. M. and Cox, R. C. Evaluation as c. tool in curriculum 
development: The IPI evaluation program , no. 5 . Chicago: 

Rand McNally, 1970. 

Scriven, M. The methodology of evaluation. Perspectives of curriculum 
evaluation, no. 1 . Chicago: Rand McNally, 1967. 

Stake, R. E. The countenance of educational evaluation. Teachers 
College Record (1967), April. 

Stake, R. E. Generalizability of program evaluation: The need for 

limits. Educational Product Report (1969) , February. 

Stuff lebeam, D. L. Evaluation as enlightenment for decision-making . 
Columbus: Evaluation Center, the Ohio State University, 1968. 





THE MONITORING ACTIVITIES OF FORMATIVE EVALUATION 






o 

ERIC 






* 




'■ *Vv-'V - r ',. v-T-t ;. „ _____ . sav 




Figure 3 
UNIT OBJECTIVES 



The Pupil: 

1. Uses terms, longer (est), taller (est), shorter (est) to 
make simple gross comparisons on the basis of length. 

2. States there are 12 inches in one foot. 

3. States there are 3 feet in one yard. 

4. States there are 36 inches in one yard. 

5. Uses 12-inch ruler to measure length of objects less than 

12 inches long to nearest inch (1/2-inch ruler markings) . 

6. Measures a given length less than 24 inches and states 
measurement in "inches" and "feet and inches." 

7. Draws line of a given length (less than 24 inches) with 
a 12-inch ruler. 









Figure A 
Activity Card 



You will need : 



L M 5b 



inch 



•foot - 



How long is 5 of your hand spans? 



Measure using M 



Measure us i ng both 



■foof 




(The actual pupil cards are colored and laminated in plastic.) 
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Figure 5 

PERCENT OF PUPILS MASTERING EACH MEASUREMENT 
OBJECTIVE IN THE LAB AND 
NON-LAB GROUPS 



Ob j ective Number 


Lab Group 
N=23 


Non-Lab 

N=23 


1 


100% 


100% 


2 


77 


52 


3 


37 


27 


4 


39 


37 


5 


63 


63 


6 


30 


17 


7 


56 


35 
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FIGURE 6 MATH LABORATORY EVALUATION OUTLINE 

Phase II 
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i. allow a pupil to make decisions as b. pupils largely select their own 

to what he will study (Goal 1, 2, 5, 9, 10) activities for study. 

. permit pupil access to materials when c. pupils secure needed materials, 

needed (Goals 1 , 2, 3, 7, 9, 1 0) 



